Image and audio processing apparatus and operating method of the same

ABSTRACT

Provided are an apparatus capable of dynamically controlling an image signal and an audio signal of content in units of content and an operating method of the apparatus. The apparatus includes a processor configured to detect characteristic information of an image signal of content based on analyzing the image signal in units of the content, detect characteristic information of an audio signal of the content based on analyzing the audio signal in the units of the content, and control the image signal in the units of the content based on the characteristic information of the image signal and the characteristic information of the audio signal to output the image signal, and control the audio signal in the units of the content based on the characteristic information of the image signal and the characteristic information of the audio signal to output the audio signal.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2018-0140097, filed on Nov. 14,2018, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an image and audio processing apparatus and anoperating method of the same, and more particularly, to an image andaudio processing apparatus capable of adaptively providing image qualityand sound depending on content characteristics and an operating methodof the image and audio processing apparatus.

2. Description of Related Art

An image and audio processing apparatus, such as a television (TV), mayprovide smart functions for users who want to enjoy various content.

However, users who want to enjoy content such as games need an image andaudio processing apparatus that adaptively supports image quality andsound according to game content, in order to more actively enjoy thegame. For example, in a case of a first person shooter (FPS) game, auser may need an image and audio processing apparatus capable ofsupporting a function of improving visibility and emphasizing a soundeffect such as a minute footstep sound even though an image may bedistorted, in order to quickly find a hidden enemy or game items hiddenin a dark area. In case of a sports game, when a visibility is improvedby distorting an image like in an FPS game, image quality ratherdegrades, and thus, a user may need an image and audio processingapparatus capable of providing a function of transferring sound intendedby a manufacturer without distorting the image. As described above,conditions of image quality and sound desired by a user vary dependingon game content because image quality and sound conditions that maymaximize a user's immersion feeling vary based on the game content,i.e., vary based on the type of game content.

Therefore, an image and audio processing apparatus that may dynamicallyprovide image quality and sound according to characteristics of content(e.g., game content) is in demand.

SUMMARY

Provided are an image and audio processing apparatus capable ofproviding image quality and sound that may maximize a user's immersionin content by dynamically controlling image signals and audio signals ofthe content by using a result of analyzing the image signal and theaudio signal in units of content (e.g., units of scenes), and anoperating method of the image and audio processing apparatus.

Further, provided are an image and audio processing apparatus capable ofproviding image quality and sound that may maximize a user's immersionin content by dynamically controlling image signals and audio signals ofthe content by using a result of analyzing the image signal and theaudio signal in units of content (e.g., units of scenes) and peripheralenvironment information, and an operating method of the image and audioprocessing apparatus.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the embodiments of the disclosure.

In accordance with an aspect of the disclosure, there is provided animage and audio processing apparatus including: a display; an audiooutput interface; a memory storing one or more instructions; and aprocessor configured to execute the one or more instructions stored inthe memory to: detect characteristic information of an image signal ofcontent based on analyzing the image signal in units of the content,detect characteristic information of an audio signal of the contentbased on analyzing the audio signal in the units of the content, andcontrol the image signal in the units of the content based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal to output the image signal through thedisplay, and control the audio signal in the units of the content basedon the characteristic information of the image signal and thecharacteristic information of the audio signal to output the audiosignal through the audio output interface.

The units of content may be units of scenes.

The processor may be further configured to execute the one or moreinstructions to: control at least one of a black equalizer function or acontrast and color saturation function with respect to the image signalin the units of the scenes, based on the characteristic information ofthe image signal and the characteristic information of the audio signal,and control at least one of an elevation perception generation and stageexpansion function or a sound equalizer application function withrespect to the audio signal in the units of the scenes, based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal.

The characteristic information of the image signal may include at leastone of brightness information, color information, contrast information,or motion information of the image signal, and the characteristicinformation of the audio signal may include at least one of locationinformation of an audio object, voice information, music information, orsound effect information.

The image and audio processing apparatus may further include: anilluminance sensor configured to detect illuminance information withrespect to outside of the image and audio processing apparatus, and theprocessor may be further configured to execute the one or moreinstructions to, based on the image signal being controlled based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal, control at least one of brightness,contrast, or color saturation of the image signal based on theilluminance information.

The processor may be further configured to execute the one or moreinstructions to, based on a learning model that includes one or moreneural networks, analyze the image signal and the audio signal of thecontent in the units of the scenes, detect the characteristicinformation of the image signal and the characteristic information ofthe audio signal, and control the image signal and the audio signal.

In accordance with an aspect of the disclosure, there is provided anoperating method of an image and audio processing apparatus, theoperating method including: analyzing, by the image and audio processingapparatus, an image signal of content in units of the content anddetecting characteristic information of the image signal; analyzing, bythe image and audio processing apparatus, an audio signal of the contentin the units of the content and detecting characteristic information ofthe audio signal; and controlling, by the image and audio processingapparatus, the image signal in the units of the content to output theimage signal through a display of the image and audio processingapparatus, based on the characteristic information of the image signaland the characteristic information of the audio signal; and controlling,by the image and audio processing apparatus, the audio signal in theunits of the content to output the audio signal through an audio outputinterface of the image and audio processing apparatus, based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal.

The units of content may be units of scenes.

The operating method may further include: controlling at least one of ablack equalizer function, or a contrast and color saturation functionwith respect to the image signal in the units of the scenes, based onthe characteristic information of the image signal and thecharacteristic information of the audio signal; and controlling at leastone of an elevation perception generation and stage expansion functionor a sound equalizer application function with respect to the audiosignal in the units of the scenes, based on the characteristicinformation of the image signal and the characteristic information ofthe audio signal.

The characteristic information of the image signal may include at leastone of brightness information, color information, contrast information,or motion information of the image signal, and the characteristicinformation of the audio signal may include at least one of locationinformation of an audio object, voice information, music information, orsound effect information.

The operating method may further include, based on the image signalbeing controlled based on the characteristic information of the imagesignal and the characteristic information of the audio signal,controlling at least one of brightness, contrast, or color saturation ofthe image signal based on illuminance information with respect tooutside of the image and audio processing apparatus, the illuminanceinformation being detected by the image and audio processing apparatus.

In accordance with an aspect of the disclosure, there is provided acomputer-readable recording medium having recorded thereon programinstructions which are executable by a processor for performing anoperating method of a image and audio processing apparatus, theoperating method including: analyzing, by the image and audio processingapparatus, an image signal of content in units of the content anddetecting characteristic information of the image signal; analyzing, bythe image and audio processing apparatus, an audio signal of the contentin the units of the content and detecting characteristic information ofthe audio signal; and controlling, by the image and audio processingapparatus, the image signal in the units of the content to output theimage signal through a display of the image and audio processingapparatus, based on the characteristic information of the image signaland the characteristic information of the audio signal; and controlling,by the image and audio processing apparatus, the audio signal in theunits of the content to output the audio signal through an audio outputinterface of the image and audio processing apparatus, based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a diagram of an image and audio processing apparatus accordingto an embodiment;

FIG. 2 is a block diagram of an image and audio processing apparatusaccording to an embodiment;

FIG. 3 is a block diagram of a processor of FIG. 2, according to anembodiment;

FIG. 4 is a block diagram of an image signal controller of FIG. 3,according to an embodiment;

FIG. 5 is a block diagram of an audio signal controller of FIG. 3,according to an embodiment;

FIG. 6 is a flowchart illustrating operations of an image and audioprocessing apparatus according to an embodiment;

FIG. 7 is a block diagram of an image and audio processing apparatusaccording to another embodiment;

FIG. 8 is a diagram showing an example of a final gain value of a blackequalizer based on a gain value of the black equalizer and illuminanceinformation, according to an embodiment;

FIG. 9 is a flowchart illustrating operations of an image and audioprocessing apparatus according to another embodiment; and

FIG. 10 is a block diagram of an image and audio processing apparatusaccording to another embodiment.

DETAILED DESCRIPTION

Hereinafter, one or more embodiments will be described in detail withreference to accompanying drawings to the extent that one of ordinaryskill in the art would be able to carry out the disclosure. However, thedisclosure may be implemented in various manners, and is not limited toone or more embodiments described herein. In addition, componentsirrelevant with the description are omitted in the drawings for cleardescription, and like reference numerals are used for similar componentsthroughout the entire specification.

The terms used in the disclosure are selected from among common termsthat are currently widely used in consideration of their functions inthe disclosure, but the terms may be different according to an intentionof one of ordinary skill in the art, a precedent, or the advent of newtechnology. Therefore, the terms used in the disclosure are not merelydesignations of the terms, but the terms are defined based on themeaning of the terms and content throughout the disclosure.

The terms used in the specification are merely used to describeparticular embodiments, and are not intended to limit the disclosure. Anexpression used in the singular encompasses the expression of theplural, unless it has a clearly singular meaning in the context.Throughout the specification, it will be understood that when an elementis referred to as being “connected” to another element, it may be“directly connected” to the other element or “electrically connected” tothe other element with intervening elements therebetween. It will befurther understood that when a part “includes” or “comprises” anelement, unless otherwise defined, the part may further include otherelements, not excluding the other elements.

Throughout the specification, in particular, in claims, “the” or othersimilar referring expressions may refer to both a singular form and aplural form. Also, if there is no description explicitly referring toorders of operations included in methods according to the disclosure,the operations may be performed in an appropriate order. The disclosureis not limited to the described orders of operations.

Phrases such as “in some embodiments” or “in one embodiment” throughoutthe specification may not necessarily denote the same embodiment of thedisclosure.

Some embodiments may be represented as functional block structures,various processing stages and/or various processing operations. Some orall of the functional blocks may be realized by any number of hardwareand/or software components configured to perform the specifiedfunctions. For example, the functional blocks of the disclosure may berealized by one or more microprocessors or circuit structures forperforming a predetermined function. In addition, for example, thefunctional blocks of the disclosure may be implemented with anyprogramming or scripting language. The functional blocks may beimplemented in algorithms that are executed on one or more processors.Also, the disclosure may employ any number of conventional techniquesfor electronics configuration, signal processing and/or, data processingand the like. The words “mechanism,” “element,” “means,” and“configuration” are used broadly and are not limited to mechanical orphysical components.

In addition, the connecting lines, or connectors shown in the variousfigures presented are intended to represent exemplary functionalrelationships and/or physical or circuit couplings between the variouselements. It should be noted that connections between elements by manyalternative or additional functional relationships, physical connectionsor circuit connections may be present in a practical device.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

FIG. 1 shows an example, in which an image and audio processingapparatus 100 according to an embodiment outputs an image signal and anaudio signal that are specialized according to content characteristics(e.g., scene characteristics) obtained by analyzing image and audiosignals of content in units of content (e.g., units of scenes).

Referring to FIG. 1, the image and audio processing apparatus 100 mayinclude, but is not limited to, a large screen TV, e.g., may include anelectronic device including a display. For example, the image and audioprocessing apparatus 100 may be implemented as various electronicdevices such as a mobile phone, a tablet PC, a digital camera, acamcorder, a laptop computer, a desktop PC, an e-book terminal, adigital broadcasting terminal, a personal digital assistant (PDA), aportable multimedia player (PMP), a navigation device, an MP3 player, awearable device, a gaming display monitor, etc. Also, the image andaudio processing apparatus 100 may be a fixed type or a portable type,and may include a digital broadcasting receiver capable of receivingdigital broadcast.

The image and audio processing apparatus 100 may be implemented as aflat panel display apparatus, and moreover may be implemented as acurved display apparatus having a curvature or a flexible displayapparatus capable of adjusting curvature thereof. An output resolutionof the image and audio processing apparatus 100 may include, forexample, high definition (HD), full HD, ultra HD, or higher resolutionthan ultra HD.

The image and audio processing apparatus 100 may be controlled by acontrol device 101. The control device 101 may be implemented as varioustypes for controlling the image and audio processing apparatus 100,e.g., a remote controller, a mobile phone, or a game console.

Also, the control device 101 may control the image and audio processingapparatus 100 via a short range wireless communication includinginfrared ray or Bluetooth. The control device 101 may control the imageand audio processing apparatus 100 via a wired communication.

The control device 101 may control functions of the image and audioprocessing apparatus 100 by using at least one of keys (includingbuttons) thereof, a touch pad, a microphone (not shown) for receivingvoice of a user, or a sensor (not shown) for sensing motions of thecontrol device 101. The control device 101 may be configured as acontrol pad exclusive for games.

The control device 101 may include a power on/off button for turning onor turning off the image and audio processing apparatus 100. Also, thecontrol device 101 may control the image and audio processing apparatusto perform one or more of change channels, adjust volume, select aterrestrial broadcast/cable broadcast, select a satellite broadcast, orset environment information of the image and audio processing apparatus100 according to a user input.

In addition, when the control device 101 is a control pad that isexclusive for games or a mobile phone, the control device 101 maytransfer game content to the image and audio processing apparatus 100and control reproduction of the game content in the image and audioprocessing apparatus 100.

In the specification, a term “user” denotes a person who controlsfunctions or operations of the image and audio processing apparatus 100by using the control device 101, and may include, but is not limited to,a gamer, a viewer, a manager, or an installation engineer.

The image and audio processing apparatus 100 according to the embodimentmay analyze image signals and audio signals of content that is beingreproduced in units of the content (e.g., units of scenes) regardless ofthe control from the control device 101, in order to respectively detectcharacteristic information of the image signals and audio signals.

The characteristic information of the image signal detected by the imageand audio processing apparatus 100 may include, for example, at leastone piece of brightness information, color information, contrastinformation, or motion information, but is not limited thereto. Thecharacteristic information of the audio signal detected by the image andaudio processing apparatus 100 may include at least one piece oflocation information, voice information, music information, or soundeffect information of an audio object, but is not limited thereto.

The image and audio processing apparatus 100 may control the imagesignal in units of scenes by using the detected characteristicinformation of the image and audio signals, and may cause the display todisplay the controlled image signal thereon. Through the above process,the image signal displayed on the image and audio processing apparatus100 denotes an image signal having image quality specialized based oncharacteristics of each scene.

The image and audio processing apparatus 100 may control an audio signalin units of scenes by using the detected image signal and audioinformation, and may output the controlled audio signal therethrough.Through the above process, the audio signal output from the image andaudio processing apparatus 100 may denote an audio signal having soundthat is specialized based on characteristics of each scene.

Also, the image and audio processing apparatus 100 may control at leastone of brightness, contrast, or color saturation of the image signal inunits of scenes by using illuminance information detected from anillumination 102, such as, a light source 102, around the image andaudio processing apparatus 100. To do this, the image and audioprocessing apparatus 100 may include an illuminance sensor.

FIG. 2 is a block diagram of an image and audio processing apparatus 200according to an embodiment.

The image and audio processing apparatus 200 of FIG. 2 may be an exampleof the image and audio processing apparatus 100 of FIG. 1. Referring toFIG. 2, the image and audio processing apparatus 200 may include amemory 210, a processor 220, a display 230, and an audio outputinterface 240. However, the image and audio processing apparatus 200 mayinclude more elements than those of FIG. 2, and is not limited to theabove example. For example, the image and audio processing apparatus 200may further include a content communication interface that obtainscontent and/or a user input interface that obtains a user input.

The memory 210 according to the embodiment may store programs forprocessing and controlling the processor 220. The memory 210 may storedata (e.g., content) input to or output from the image and audioprocessing apparatus 200. The memory 210 may store information used toprocess and control the processor 220. The information used to processand control the processor 220 may include, for example, settinginformation of the image and audio processing apparatus 200 forproviding image quality and sound specialized in units of scenes.

The setting information of the image and audio processing apparatus 200may include, for example, information used for controlling at least oneof the image signal and the audio signal of the scene based on eachand/or combinations of brightness information of the image signal,contrast and/or color saturation information of the image signal, motioninformation of the image signal, location information of an audio objectincluded in the audio signal, voice information included in the audiosignal, music information included in the audio signal, and/or soundeffect information included in the audio signal.

The setting information of the image and audio processing apparatus 200may include, for example, information for determining a kind of soundeffect (e.g., a gunshot sound, a crowd sound, a car engine sound, etc.)included in the audio signal of the unit (e.g., scene), information fordetermining brightness of the image signal of the unit (e.g., scene)(e.g., a threshold value of the number of pixels determined as a darkimage), and information about a method of controlling the image signaland the audio signal of the unit (e.g., scene) based on characteristicinformation of the audio signal and the image signal of the unit (e.g.,scene), but is not limited thereto.

For example, based on the characteristic information of the image signalof the scene including information indicating that the image signal isdark and the characteristic information of the audio signal of the sceneincluding a gun sound effect (e.g., based on the characteristicinformation of the image signal of the scene being identified as a scenefrom an First Person Shooters (FPS) game), the information about themethod of controlling the image signal and the audio signal of the unit(e.g., scene) may include information for controlling the image signalso that the processor 220 increases a gain of a gain function of a blackequalizer with respect to the image signal and emphasizes the contrastand color saturation for improving visibility of the image signal andcontrolling the audio signal so that a volume of background sound may beup by applying a sound equalizer.

For example, based on the characteristic information of the image signalof the unit (e.g., scene) including information indicating that theimage signal is bright and the characteristic information of the audiosignal of the unit (e.g., scene) including a gun sound effect (e.g.,based on the characteristic information of the image signal of the scenebeing identified as a scene from an FPS game), the information about themethod of controlling the image signal and the audio signal of the unit(e.g., scene) may include information for controlling the image signalso that the processor 220 reduces a gain of a gain function of a blackequalizer with respect to the image signal and emphasizes only thecontrast and color saturation and controlling the audio signal so thatbackground sound of the unit (e.g., scene) may be clearly heard byapplying a sound equalizer.

For example, based on the characteristic information of the image signalof the unit (e.g., scene) including information indicating that theimage signal is bright and the characteristic information of the audiosignal of the scene including a voice signal of one person (e.g., basedon the characteristic information of the image signal of the unit (e.g.,scene) being identified as a unit (e.g., scene) from a general game),the information about the method of controlling the image signal and theaudio signal of the scene may include information for controlling theimage signal so that the processor 220 does not operate the blackequalizer with respect to the image signal of the scene and reducesemphasizing a level of at least one of the contrast and/or colorsaturation and controlling the audio signal so that a sound effect orbackground sound of the unit (e.g., scene) may be emphasized by applyinga sound equalizer.

For example, based on the characteristic information of the image signalof the unit (e.g., scene) including color information indicating grasscolor of a playground and the characteristic information of the audiosignal of the unit (e.g., scene) including sound of the audience (e.g.,based on the characteristic information of the image signal of the unit(e.g., scene) being identified as a unit of content (e.g., scene) from asports game), the information about the method of controlling the imagesignal and the audio signal of the unit (e.g., scene) may includeinformation for controlling the image signal so that the processor 220does not operate the black equalizer with respect to the image signal ofthe unit (e.g., scene) and may emphasize at least one of contrast and/orcolor saturation in order to make the grass color clear and/ordiscriminate a color of a uniform and controlling the audio signal sothat a surround sound experience of feeling like being surrounded byroaring of the audience by at least one of: expanding a sound image of ahorizontal plane, expanding a stage, or applying the sound equalizer.

For example, based on the characteristic information of the image signalof the unit (e.g., scene) including information indicating a relativelylarge amount of motion information and a bright screen and thecharacteristic information of the audio signal of the unit (e.g., scene)including a sound of a car engine (e.g., based on the characteristicinformation of the image signal of the unit (e.g., scene) beingidentified as a unit of content (e.g., scene) from a racing game), theinformation about the method of controlling at least one of the imagesignal or the audio signal of the unit (e.g., scene) may include atleast one of: information for controlling the image signal so that theprocessor 220 does not operate the black equalizer with respect to theimage signal of the unit (e.g., scene) or emphasizing at least one ofcontrast and/or color saturation in order to make the cars and/orbackground track clear or controlling the audio signal so that a lowbandwidth signal such as the car engine sound may be emphasized byapplying the sound equalizer.

For example, based on the characteristic information of the image signalof the unit (e.g., scene) including information indicating a lot ofmotion information and a dark screen (e.g., based on the characteristicinformation of the image signal of the unit (e.g., scene) beingidentified as a unit of content (e.g., scene) of passing through atunnel) and the characteristic information of the audio signal of theunit (e.g., scene) including a sound of a car engine (e.g., based on thecharacteristic information of the audio signal of the unit (e.g., scene)being identified as a unit of content (e.g., scene) from a racing game),the information about the method of controlling at least one of theimage or audio signal of the unit (e.g., scene) may include informationfor controlling the image signal so that the processor 220 increasesbrightness of the screen (or improves visibility) by applying a blackequalizer function to the image signal of the unit (e.g., scene) and/oremphasizes at least one of the contrast and/or color saturation to makethe cars and/or background track clear, and controlling the audio signalso as to emphasize the low-bandwidth signal such as the car engine soundby applying the sound equalizer.

The memory 210 may include a storage medium of at least one type of aflash memory type, a hard disk type, a multimedia card micro type, acard type memory (e.g., a Secure Digital (SD) memory or an eXtremeDigital (XD) memory, etc.), random access memory (RAM), a static RAM(SRAM), a read-only memory (ROM), an electrically erasable programmableROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magneticdisk, or an optical disk.

The processor 220 according to the embodiment may execute one or moreinstructions stored in the memory 210 to respectively analyze an imagesignal of content and an audio signal of the content in units of sceneswith respect to the content to be reproduced through the display 230 andthe audio output interface 240 and detect characteristic information ofthe image and audio signals. The characteristic information of the imagesignal and the characteristic information of the audio signal aredescribed above with reference to FIG. 1.

The processor 220 may control the image signal of the unit (e.g., scene)by using at least one of: the characteristic information of the imagesignal or the audio signal, and then, the specialized image signal maybe displayed on the display 230. The processor 220 may control the audiosignal of the unit (e.g., scene) by using the characteristic informationof at least one of: the image signal or audio signal, and then thespecialized audio signal may be output through the audio outputinterface 240.

For example, based on the audio signal being analyzed in units of scenesincluding a gunshot sound and the image signal being analyzed in unitsof scenes being identified as a dark scene, the processor 220 maycontrol the image and audio signals of the unit (e.g., scene) to havethe image quality and sound specialized for FPS game content. Forexample, the processor 220 may control the image signal and the audiosignal of the unit (e.g., scene), and thus game items included in theunit (e.g., scene) may have high visibility and a bandwidth of a certainsound effect such as a footstep sound may be reinforced.

Based on identifying that the audio signal analyzed in units of scenesincludes the car engine sound and the image signal analyzed in units ofscenes includes a large amount of motion information detected from theimage signal, the processor 220 may control the image and audio signalsof the unit (e.g., scene) to have the image quality and sound that arespecialized for racing game content. For example, the processor 220 maycontrol the image and audio signals of the unit (e.g., scene) in orderto reinforce the bandwidth of the sound effect of the car engine soundwhile improving color saturation of the image signal. The large amountof motion information may correspond to an amount of motion informationthat is above a predetermined threshold amount of motion.

Based on identifying that the audio signal is analyzed in units ofscenes includes a voice signal and the image signal analyzed in units ofscenes indicates a bright scene, the processor 220 may control the imageand audio signals of the unit (e.g., scene) to have image quality andsound that are specialized for general game content. For example, theprocessor 220 may control the image and audio signals of the unit (e.g.,scene) in order to reinforce the bandwidth of the sound effect and/orbackground sound included in the unit (e.g., scene) without controllingcontrast and color saturation of the image signal of the unit (e.g.,scene).

In the embodiment, the processor 220 may store one or more instructionsin a memory (not shown) that is built therein, and may perform the aboveoperations by executing one or more instructions stored in the memory(not shown) built therein. That is, the processor 220 may perform acertain operation by executing at least one instruction or programstored in the memory (not shown) included in the processor 220 or thememory 210.

Also, in the embodiment, the processor 220 may include a graphicprocessing unit (GPU) (not shown) for processing graphics correspondingto the image. The processor (not shown) may be implemented as a systemon chip (SoC) combining a core (not shown) with a GPU (not shown). Theprocessor (not shown) may include at least one of: a single core, adual-core, a triple-core, a quad-core, or a multiple-core processor.

The processor 220 may control overall operations of the image and audioprocessing apparatus 200. For example, the processor 220 may control thedisplay 230 and the audio output interface 240 by executing one or moreinstructions stored in the memory 210.

Also, FIG. 2 shows one processor 220, but there may be a plurality ofprocessors (not shown). In this case, each of the operations performedby the image and audio processing apparatus 200 according to theembodiment may be executed by at least one of the plurality ofprocessors (not shown). For ease of understanding, the specificationdiscusses the processor, which could refer to one processor or aplurality of processors that perform the functions of the one processor.

The processor 220 may execute one or more instructions stored in thememory 210, and detect characteristic information of the image signaland characteristic information of the audio signal by analyzing theimage signal and the audio signal of the content in units of scenes byusing a learning model using one or more neural networks stored in thememory 210, and may control at least one of the image or audio signal ofthe unit (e.g., scene) by using the detected characteristic informationof at least one of the image or audio signals.

The neural network may include a set of algorithms that may detectcharacteristic information respectively from at least one of the imagesignal or the audio signal input to the neural network based onartificial intelligence (AI) and train so as to learn a method ofcontrolling the image signal and the audio signal in units of scenesbased on the detected characteristic information. For example, theneural network may perform recognition (e.g., image recognition,voice/sound recognition) so as to recognize characteristic informationfrom at least one of the image and/or audio signals and learn a methodof controlling at least one of the image and audio signals, based onsupervised learning that has at least one of the image signal or theaudio signal in units of scenes as input values, and unsupervisedlearning that finds a pattern for recognizing the characteristicinformation from the image and audio signals by self-learningkinds/types of data that are required to detect the characteristicinformation respectively from the image and/or audio signals without anysupervision. Also, for example, the neural network may detect thecharacteristic information from the image and/or audio signals in unitsof scenes and learn the method of controlling the image and audiosignals in units of scenes, by using reinforcement learning that usesfeedback about whether the image and/or audio signals are controlledappropriately based on the detected characteristic information of theimage and/or audio signals.

Also, the neural network may perform operations for inference andprediction according to artificial intelligence (AI) technology. Indetail, the neural network may be a deep neural network (DNN) thatperforms operations through a plurality of layers. The neural networkmay be classified as a DNN based on a number of the plurality of layers,that is, a depth of the neural network performing the operationincreases, according to the number of internal layers performing theoperation. In addition, the DNN operation may include a convolutionneural network (CNN) operation. That is, the processor 220 may implementa model for detecting characteristic information of at least one of theimage signal and the audio signal through the neural network of theexample, and the implemented model may be learned by using learningdata. In addition, the at least one of the image signal and the audiosignal in units of scenes in the content to be reproduced may beanalyzed or classified by using a learned model to detect thecharacteristic information of at least one of the image signal and theaudio signal of the unit (e.g., scene).

For example, the processor 220 may detect characteristic information byrespectively analyzing at least one of the image signal and the audiosignal of the content to be reproduced in units of scenes, by using alearning model using one or more neural networks. For example, theprocessor 220 may detect at least one of brightness information,contrast information, or color information of an image signal in a sceneand/or at least one of location information of an audio object, voiceinformation, music information, or sound effect information of the audiosignal, by performing operations through the DNN.

The processor 220 may control at least one of: the image signal and theaudio signal in units of scenes based on the characteristic informationof at least one of: the image and audio signals detected by using theneural network. For example, the processor 220 may control at least oneof a black equalizer function, contrast, or color saturation withrespect to the image signal and/or at least one of a function ofgenerating elevation perception and expanding stage with respect to theaudio signal, or a sound equalizer function with respect to the audiosignal in units of scenes, by using the characteristic information of atleast one of the image and audio signals detected by using the neuralnetwork.

The processor 220 may control the display 230 to output (display) theimage signal that is controlled in units of scenes. According to theembodiment, the processor 220 may control the audio output interface 240to output the audio signal that is controlled in units of scenes.

The display 230 according to the embodiment may display the image signalthat is controlled in units of scenes. For example, based on a scenebeing from an FPS game and including a dark image signal, the display230 may display an image signal for providing high visibility of gameitems included in the scene. Based on a scene being from a sports game,the display 230 may display an image signal that is controlled toemphasize at least one of the contrast and/or color saturation so thatthe uniform(s) of each team may be clearly distinguished.

When the display 230 is implemented as a touch screen, the display 230may be also used as an input device, as well as the output device. Forexample, the display 230 may include at least one of a liquid crystaldisplay, a thin film transistor-liquid crystal display, an organiclight-emitting diode, a flexible display, a three-dimensional (3D)display, or an electrophoretic display. In addition, according toimplementation type of the image and audio processing apparatus 200, theimage and audio processing apparatus 200 may include two or moredisplays 230.

The audio output interface 240 may output an audio signal that iscontrolled in units of scenes according to control of the processor 220.The audio output interface 240 may include at least one of a speaker, aheadphone output terminal, or a Sony/Philips digital interface (S/PDIF)output terminal, but is not limited thereto.

For example, based on a scene being from an FPS game and including adark image signal, the audio output interface 240 according to theembodiment may output an audio signal obtained by generating anelevation perception or expanding a sound stage of the audio signal(e.g., footstep sound) included in the unit (e.g., scene). Also, basedon the scene being from a sports game, the audio output interface 240may output an audio signal obtained by generating an elevationperception and expanding a sound image of a horizontal plane of thesound effect (e.g., roaring sound) included in the audio signal of theunit (e.g., scene).

FIG. 3 is a block diagram of the processor 220 of FIG. 2.

In FIG. 3, the processor 220 of FIG. 2 includes an image signal analyzer310, an audio signal analyzer 320, an image signal controller 330, andan audio signal controller 340, but elements included in the processor220 are not limited thereto.

The image signal analyzer 310 may analyze the image signal in units ofscenes. For example, the image signal analyzer 310 may identify whetherthe scene is dark or bright according to a histogram distribution ofblack area, and provide an identification result to at least one of theimage signal controller 330 and the audio signal controller 340. Theimage signal analyzer 310 may use a threshold value in the number ofpixels, which indicates the dark region, stored in the memory 210, inorder to identify whether the scene is dark or bright according to thehistogram distribution of the black area.

The image signal analyzer 310 may perform a histogram analyzation anddetail analyzation of a black region in each local area, as well as thehistogram analysis of the black region in units (e.g., frame units withrespect to one scene), in order to more accurately analyze the imagesignal in units of scenes.

The audio signal analyzer 320 may analyze the audio signal in units ofscenes. For example, the audio signal analyzer 320 may detect at leastone of location information of an audio object, voice information, musicinformation, or sound effect information included in the audio signal inunits of scenes. The audio signal analyzer 320 may use information aboutthe sound effect stored in the memory 210 in order to detect the soundeffect. The information about the sound effect stored in the memory 210may include information for identifying a kind/type of the sound effect.The audio signal analyzer 320 may provide a result of analyzing theaudio signal to at least one of the image signal controller 330 and theaudio signal controller 340.

The image signal controller 330 may control at least one of a blackequalizer function, contrast, or color saturation function with respectto the image signal in units of scenes, by using the image signalanalysis result provided from at least one of the image signal analyzer310 and the audio signal analysis result provided from the audio signalanalyzer 320. Accordingly, the image signal controller 330 may output animage signal having image quality specialized for the scene.

FIG. 4 is a block diagram of the image signal controller 330 of FIG. 3.

Referring to FIG. 4, the image signal controller 330 may include atleast one of a black equalizer dynamic controller 410 and acontrast/color saturation controller 420.

The black equalizer dynamic controller 410 may control brightness of theimage signal by using characteristic information of at least one of aninput image signal or an audio signal, in order to improve blackvisibility in a dark area in the input image signal.

The contrast/color saturation controller 420 may control contrast and/orcolor saturation of the input image signal by using the characteristicinformation of at least one of: the input image signal and or the audiosignal.

For example, based on information indicating a sound effectcorresponding to a gun shoot sound being obtained from the audio signalanalyzer 320 and information indicating that the unit (e.g., scene) is adark unit (e.g., scene) being provided from the image signal analyzer310, the image signal controller 330 may control brightness of the imagesignal by increasing a gain value (e.g., a gain value of the blackequalizer) of the black equalizer dynamic controller 410 so that thedark unit (e.g., scene) becomes bright, and may control the image signalby using the contrast/color saturation controller 420 so that at leastone of the contrast and color saturation of the image signal may beemphasized.

Based on information about a sound effect of a car engine sound beingprovided from the audio signal analyzer 320 and motion information (arelatively large amount of motion information) indicating a fast movingscene and information indicating a bright scene are provided from theimage signal analyzer 310, the image signal controller 330 may controlthe image signal by using the at least one of contrast/color saturationcontroller 420 so as to emphasize the at least one of contrast/colorsaturation without using the black equalizer dynamic controller 410.

For example, based on information about a sound effect of a car enginesound being provided from at least one of the audio signal analyzer 320and motion information (a large amount of motion information) indicatinga fast moving scene and information indicating a dark scene beingprovided from the image signal analyzer 310, the image signal controller330 may control the image signal to brighten the screen by using theblack equalizer dynamic controller 410 and may control the image signalto emphasize the color of the image signal by using the contrast/colorsaturation controller 420.

For example, based on the sound effect and background sound providedfrom the audio signal analyzer 320 being identified as general gamecontent and information indicating a dark scene being provided from theimage signal analyzer 310, the image signal controller 330 may controlthe image signal so as not to excessively emphasize at least one of: thecontrast and color saturation without using the black equalizerfunction.

The audio signal controller 340 of FIG. 3 may control at least one of afunction of generating elevation perception and expanding stage withrespect to the audio signal, or a sound equalizer function with respectto the audio signal in units of scenes, by using the result of analyzingthe image signal provided from the image signal analyzer 310 and theresult of analyzing the audio signal provided from the audio signalanalyzer 320. Accordingly, the audio signal controller 340 may output anaudio signal having sound specialized for the scene (e.g., sound qualityspecialized for the scene).

FIG. 5 is a block diagram of the audio signal controller 340 of FIG. 3.

Referring to FIG. 5, the audio signal controller 340 may include, but isnot limited to, at least one of: an elevation perception generator andstage expander 510 and a sound equalizer applier 520.

The elevation perception generator and stage expander 510 may at leastone of reinforce motion of an audio object or may expand a sound stage.For example, based on an audio signal of a scene including a soundeffect, the elevation perception generator and stage expander 510 maygenerate an elevation perception of the sound effect and may expand ahorizontal plane sound image. For example, based on the audio signal ofthe unit (e.g., scene) including roaring sound in a stadium, theelevation perception generator and stage expander 510 may performexpansion of a horizontal sound image and expansion of a stage withrespect to the roaring sound to provide a user with a sound experienceas if the user is surrounded by the roaring sound. The generation ofelevation perception and the expansion of the horizontal plane soundimage may be realized by a head-related transfer function (HRTF) that iswidely used in audio signal processing field, but is not limitedthereto. The expansion of the stage may be implemented in a manner ofadding opposite phase signals of different audio channel to eachchannel, but is not limited thereto.

The sound equalizer applier 520 may apply a discriminative equalizer toeach unit (e.g., scene) by using an input audio signal. For example,based on an audio signal of a unit (e.g., scene) including a car enginesound, the sound equalizer applier 520 may be controlled to reinforce alow bandwidth component, and then, a realistic car engine sound may beprovided to the user. For example, based on an audio signal of a unit(e.g., scene) including music, the sound equalizer applier 520 may becontrolled to reinforce both or at least one of: low-bandwidth andhigh-bandwidth components and instrument sound expanding throughoutentire or partial bandwidths may be provided to the user.

Also, for example, based on information indicating a sound effect of agunshot sound being provided from the audio signal analyzer 320 andinformation indicating that the scene is a dark scene being providedfrom the image signal analyzer 310, the audio signal controller 340 maycontrol the audio signal to reinforce background sound by using thesound equalizer applier 520.

Based on information about a sound effect of a car engine sound beingprovided from the audio signal analyzer 320 and motion informationindicating a fast moving scene and information indicating a bright scenebeing provided from the image signal analyzer 310, the audio signalcontroller 340 may control the audio signal to reinforce a low-bandwidthcomponent such as the car engine sound by using the sound equalizerapplier 520.

Based on information about at least one of: a sound effect of a carengine sound being provided from the audio signal analyzer 320 andmotion information indicating a fast moving unit of content (e.g.,scene) and information indicating a dark unit of content (e.g., scene)being provided from the image signal analyzer 310, the audio signalcontroller 340 may control the audio signal to reinforce a low-bandwidthcomponent such as the car engine sound by using the sound equalizerapplier 520.

Based on information indicating detection of voice being provided fromthe audio signal analyzer 320 and information indicating a dark scenebeing provided from the image signal analyzer 310, the audio signalcontroller 340 may emphasize at least one of a sound effect and abackground sound by expanding at least one of the sound effect and thebackground sound by using the elevation perception generator and stageexpander 510 and may control the audio signal to reinforce thelow-bandwidth component and the high-bandwidth component by using thesound equalizer applier 520.

Also, for example, based at least one of: obtained characteristicinformation of the image signal indicating a dark unit of content (e.g.,scene) and received characteristic information of the audio signalincluding a footstep sound, the sound equalizer applier 520 mayreinforce a bandwidth of a certain sound effect such as the footstepsound to provide clear footstep sound to the user.

For example, based on the obtained characteristic information of theimage signal including a large amount of motion information, the audiosignal controller 340 may expand a sound image of the audio objectincluded in the obtained audio signal by using the elevation perceptiongenerator and stage expander 510 and may control the audio signal toreinforce the sound effect included in the audio signal by using thesound equalizer applier 520.

FIG. 6 is a flowchart illustrating operations of an image and audioprocessing apparatus according to an embodiment. The method (600) ofoperating the image and audio processing apparatus illustrated withreference to FIG. 6 may include operations performed in the image andaudio processing apparatus 100 or 200 according to the embodimentdescribed above with reference to FIG. 1 or FIG. 2. Therefore, indetailed operations of the method (600) of operating the image and audioprocessing apparatus, descriptions about the same operations as thoseperformed in the image and audio processing apparatus 100 or 200according to the embodiment are omitted. The method (600) of operatingthe image and audio processing apparatus of FIG. 6 shows a case in whichan image signal and an audio signal of content are respectively analyzedin units of scenes to control at least one of the image and audiosignals.

Referring to FIG. 6, in operation S610, the image and audio processingapparatus 100 may detect characteristic information of the image signalby analyzing the image signal of the content in units of scenes.Processes of analyzing the image signal and detecting characteristicinformation of the image signal in operation S610 may be the same asthose of FIGS. 1 to 5. The characteristic information of the imagesignal is described above with reference to FIG. 1.

In operation S620, the image and audio processing apparatus 100 mayanalyze an audio signal of the content in units of scenes to detectcharacteristic information of the audio signal. Processes of analyzingthe audio signal and detecting characteristic information of the audiosignal in operation S620 may be the same as those of FIGS. 1 to 5. Thecharacteristic information of the audio signal is described above withreference to FIG. 1.

In operation S630, the image and audio processing apparatus 100 maycontrol the image signal based on at least one of the characteristicinformation of the image signal and the characteristic information ofthe audio signal. The controlling of the image signal performed inoperation S630 is described above with reference to FIGS. 1 to 5.

In operation S640, the image and audio processing apparatus 100 maycontrol the audio signal based on at least one of the characteristicinformation of the image signal and the characteristic information ofthe audio signal. The controlling of the audio signal performed inoperation S640 is described above with reference to FIGS. 1 to 5.

FIG. 7 is a block diagram of an image and audio processing apparatus 700according to another embodiment.

In FIG. 7, the same elements as those of FIG. 2 are denoted by the samereference numerals. Therefore, descriptions about the elements in theimage and audio processing apparatus 700, which may be the same as thoseillustrated with reference to FIGS. 1 to 5, are omitted.

Referring to FIG. 7, the image and audio processing apparatus 700 mayfurther include an illuminance sensor 750, as compared with the imageand audio processing apparatus 200 of FIG. 2.

The illuminance sensor 750 may detect peripheral illuminance informationoutside of the image and audio processing apparatus 700. The peripheralilluminance information denotes environmental information around theimage and audio processing apparatus 700. The environmental informationaround the image and audio processing apparatus 700 is not limited tothe illuminance information. For example, when the audio outputinterface 240 of the image and audio processing apparatus 700 is aspeaker, the environmental information around the image and audioprocessing apparatus 700 may include surrounding sound (e.g., noise).The illuminance sensor 750 may indicate brightness information as amagnitude value, and detected peripheral illuminance information mayinclude brightness information of an illumination 102 around the imageand audio processing apparatus 700 detected as a magnitude value.

A memory 710 may store information about a relation between a gain valueof the black equalizer and an illuminance value. The image and audioprocessing apparatus 700 may compensate for an intensity of the blackequalizer according to a peripheral illuminance of the image and audioprocessing apparatus 700 based on the information about the relationbetween the gain value of the black equalizer and the illuminance valuestored in the memory 710, in order to provide consistent visibility withrespect to the content. That is, the image and audio processingapparatus 700 may finally determine a gain value of the black equalizerto be applied, based on a gain value of the black equalizer determinedaccording to at least one of the characteristic information of the imagesignal, the characteristic information of the audio signal, detectedilluminance information, and the above relation information stored inthe memory 710.

FIG. 8 is a graph showing an example of a final gain value of the blackequalizer based on a gain value (BE_GAIN, or gain control value) of theblack equalizer and illuminance information, according to an embodiment.In the example of FIG. 8, based on the illuminance information having apredetermined value (e.g., a value of 150 Lux), the gain value of theblack equalizer may be expressed as a linear equation.

Based on the image signal being controlled based on at least one of thecharacteristic information of the image signal and characteristicinformation of the audio signal detected by respectively analyzing atleast one of the image signal and the audio signal of the content inunits of scenes, a processor 720 may control the black equalizer byusing a final gain value of the black equalizer, which may be identifiedbased on at least one of illuminance information detected by theilluminance sensor 750 and the relation between the gain value of theblack equalizer and the illuminance information stored in the memory710.

Also, based on the audio signal being controlled based on at least oneof: characteristic information of the image signal and characteristicinformation of the audio signal detected by respectively analyzing atleast one of the image signal and the audio signal of the content inunits of scenes, the processor 720 may control the audio signal based ona brightness of the image signal that is to be controlled based on thefinal gain value of the black equalizer, which may be identified basedon at least one of illuminance information detected by the illuminancesensor 750 and the relation between the gain value of the blackequalizer and the illuminance information stored in the memory 710.

FIG. 9 is a flowchart illustrating operations of an image and audioprocessing apparatus, according to another embodiment. A method (900) ofoperating the image and audio processing apparatus illustrated withreference to FIG. 9 may include operations performed in the image andaudio processing apparatus 100 or 700 according to the embodimentdescribed above with reference to FIG. 1 or FIG. 7. Therefore, indetailed operations of the method (900) of operating the image and audioprocessing apparatus, descriptions about the same operations as thoseperformed in the image and audio processing apparatus 100 or 700according to the embodiment are omitted.

In operation S910, the image and audio processing apparatus 700 mayanalyze an image signal of the content in units of scenes to detectcharacteristic information of the image signal. Analyzing of the imagesignal of the content and detecting of the characteristic information ofthe image signal performed in operation S910 may be performed as inoperation S610. The characteristic information of the image signaldescribed in operation S910 is described above with reference to FIG. 1.

In operation S920, the image and audio processing apparatus 700 mayanalyze an audio signal of the content in units of scenes to detectcharacteristic information of the audio signal. Analyzing of the audiosignal of the content and detecting of the characteristic information ofthe audio signal performed in operation S920 may be performed as inoperation S620. The characteristic information of the audio signaldescribed in operation S920 is described above with reference to FIG.

In operation S930, the image and audio processing apparatus 700 maydetect peripheral illuminance information of the image and audioprocessing apparatus 700. Detecting of the peripheral illuminanceinformation in operation S930 may be the same as that described withreference to FIG. 7.

In operation S940, the image and audio processing apparatus 700 maycontrol the image signal based on at least one of: the characteristicinformation of the image signal, the characteristic information of theaudio signal, and the illuminance information. Controlling of the imagesignal performed in operation S940 may be the same as that describedabove with reference to FIG. 7.

In operation S950, the image and audio processing apparatus 700 maycontrol the audio signal based on at least one of: the characteristicinformation of the image signal, the characteristic information of theaudio signal, and the illuminance information. Controlling of the audiosignal performed in operation S950 may be the same as that describedabove with reference to FIG. 7.

FIG. 10 is a block diagram of an image and audio processing apparatus1000 according to another embodiment.

As shown in FIG. 10, the image and audio processing apparatus 1000 mayfurther include a tuner 1010, a processor 1020, a communicator 1030, asensor 1040, an inputter/outputter 1050, an image signal processor 1060,an audio signal processor 1070, and a user inputter 1080, in addition tothe display 230 and the audio output interface 240.

Descriptions about the display 230 and the audio output interface 240,which are already provided above with reference to FIG. 2, are omitted.

The tuner 1010 may tune and select a frequency of a channel to bereceived by the image and audio processing apparatus 1000, from amongmany wave components, through amplification, mixture, resonance, etc. ofbroadcast signal that is received wirelessly or through wires. The tuner1010 may be a television (TV) tuner. The broadcast signal may includeaudio, video, and additional information (e.g., electronic program guide(EPG) information).

The broadcast signal received by the tuner 1010 may be decoded (e.g.,audio decoding, video decoding, or additional information decoding) andmay be separated as audio, video, and/or additional information. Theseparated audio, video, and/or additional information may be controlledby the processor 1020 and stored in a memory 1090.

The image and audio processing apparatus 1000 may include one or moretuners 1010. The tuner 1010 may be implemented integrally with the imageand audio processing apparatus 1000 (all-in-one) or an additional devicehaving a tuner electrically connected to the image and audio processingapparatus 1000 (e.g., a set-top box (not shown), a tuner (not shown)connected to the inputter/outputter 1050).

The communicator 1030 may connect the image and audio processingapparatus 1000 to an external device (e.g., an audio device, a server, acontent providing device, etc.) according to control of the processor1020. The communicator 1030 may be a communication interface. Theprocessor 1020 may at least one of: transmit/receive content to/from theexternal device connected through the communicator 1030, download anapplication from the external device, or perform web-browsing.

The communicator 1030 may be a communication interface. The communicator1030 may include one or more of a wireless LAN, a Bluetooth, or a wiredEthernet, in correspondence with the performance and structure of theimage and audio processing apparatus 1000. Alternatively, thecommunicator 1030 may include a combination of the wireless LAN,Bluetooth, and wired Ethernet. The communicator 1030 may receive acontrol signal from a control device 101 according to the control of theprocessor 1020. The control signal may be implemented as a Bluetoothtype, a radio frequency (RF) signal type, or a Wi-Fi type.

The communicator 1030 may further include another near fieldcommunicator (e.g., near field communication (NFC) interface (notshown), and Bluetooth Low Energy (BLE) interface (not shown)), inaddition to the Bluetooth.

The communicator 1030 according to the embodiment may obtain ainformation regarding a learning model based on one or more neuralnetworks from an external server. Also, the communicator 1030 may obtainnew information from the external server with a preset cycle, in orderto update information that may be used to analyze at least one of theimage signal and the audio signal stored in the memory 1090.

The sensor 1040 may detect at least one of: a voice of the user, animage of the user, an interaction of the user, and illuminance aroundthe image and audio processing apparatus 1000, and may include at leastone of: a microphone 1041, a camera 1042, a light receiver 1043, or anilluminance sensor 1044.

The microphone 1041 may obtain a voice uttered by the user. Themicrophone 1041 may convert the obtained voice into an electrical signaland may output the electrical signal to the processor 1020.

The camera 1042 may obtain an image (e.g., successive frames)corresponding to a motion of the user including gesture, within a camerarecognition range.

The light receiver 1043 may obtain an optical signal (including acontrol signal) from the control device 101. The light receiver 1043 mayreceive an optical signal corresponding to a user input (e.g., touch,push, touch gesture, voice, or motion) from the control device 101. Theprocessor 1020 may detect a control signal from the optical signal.

The light receiver 1043 according to the embodiment may receive anoptical signal corresponding to a user input regarding contentreproduction, from the control device 101.

The illuminance sensor 1044 may detect illuminance information based onilluminance of the illumination 102 outside of the image and audioprocessing apparatus 1000 like the illuminance sensor 750 of FIG. 7, buta detection range of the illuminance sensor 1044 is not limited thereto.For example, the illuminance sensor 1044 may detect illuminance of alight (e.g., sunlight) around the image and audio processing apparatus1000.

The inputter/outputter 1050 is controlled by the processor 1020 toobtain a video (e.g., moving pictures, etc.), audio (e.g., voice, music,etc.), and additional information (e.g., EPG, etc.) from the outside ofthe image and audio processing apparatus 1000. The inputter/outputter1050 may include at least one of a high-definition multimedia interface(HDMI) port 1051, a component jack 1052, a PC port 1053, or a universalserial bus (USB) port 1054. The inputter/outputter 1050 may include atleast one of or a combination of the HDMI port 1051, the component jack1052, the PC port 1053, and the USB port 1054.

A memory 1090 according to the embodiment may store programs forprocessing and controlling the processor 1020, and data input to oroutput from the image and audio processing apparatus 1000. Also, thememory 1090 may store data that is required to operate the image andaudio processing apparatus 1000. The data required to operate the imageand audio processing apparatus 1000 may include information stored inthe memory 210 as described above with reference to FIG. 2.

Also, the programs stored in the memory 1090 may be classified as aplurality of modules according to functions thereof. In detail, thememory 1090 may store one or more programs for executing a predeterminedoperation by using a neural network. For example, the one or moreprograms stored in the memory 1090 may be classified as a learningmodule 1091 and a control module 1092.

The learning module 1091 may include a learning model that learns amethod of, based on an input of content into one or more neuralnetworks, analyzing at least one of an image signal of content and anaudio signal of the content in units of scenes to detect characteristicinformation and may control at least one of the image signal and theaudio signal of the scene by using the detected characteristicinformation. The learning model may be obtained from an external server,and the learning model may be stored in the learning module 1091.

The control module 1092 may store a program that the processor 1020executes one or more instructions, and then may perform controlling ofat least one of the image signal and the audio signal according to atleast one of characteristic information of the image signal andcharacteristic information of the audio signal obtained based oninformation used to analyze at least one of the image signal and theaudio signal and a result of analyzing the image signal and the audiosignal of the content in units of scenes. For example, the controlmodule 1092 may store a program that the processor 1020, based on inputcontent, detects at least one of characteristic information of the imagesignal and characteristic information of the audio signal by comparingthe result of analyzing at least one of the image signal and the audiosignal with stored information, controls the image signal processor 1060and the display 230 to output the image signal controlled in units ofscenes and controls the audio signal processor 1070 and the audio outputinterface 240 to output the audio signal controlled in units of scenes,by providing at least one of operating condition information forcontrolling the image signal and operating condition information forcontrolling the audio signal according to the detected characteristicinformation of the image signal and audio signal to the processor 1020.

In addition, one or more programs for performing predeterminedoperations by using the neural network or one or more instructions forperforming predetermined operations by using the neural network may bestored in an internal memory (not shown) in the processor 1020.

The processor 1020 may control overall operations of the image and audioprocessing apparatus 1000 and flow of signals among internal componentsof the image and audio processing apparatus 1000, and process the data.Based on a user input being entered or a stored preset condition beingsatisfied, the processor 1020 may execute an operation system (OS) andvarious applications stored in the memory 1090.

Also, the processor 1020 may include an internal memory (not shown). Inthis case, at least one of data, programs, or instructions stored in thememory 1090 may be stored in the internal memory (not shown) of theprocessor 1020. For example, the internal memory (not shown) of theprocessor 1020 may store one or more programs for performingpredetermined operations by using the neural network, or one or moreinstructions for performing predetermined operations by using the neuralnetwork.

The image signal processor 1060 may process image data to be displayedon the display 230, and may perform various image processing operationssuch as at least one of decoding, rendering, scaling, noise filtering,frame rate conversion, and resolution conversion on the image data. Inparticular, the image signal processor 1060 may be controlled by theprocessor 1020 to perform functions corresponding to those of at leastone of the black equalizer dynamic controller 410 and the contrast/colorsaturation controller 420 shown in FIG. 4.

The display 230 may be controlled by the processor 1020 to display avideo included in the broadcast signal obtained through the tuner 1010.Also, the display 230 may display content (e.g., video) input throughthe communicator 1030 and/or the inputter/outputter 1050. The display230 may output the image stored in the memory 1090 according to controlof the processor 1020. The display 230 may be controlled by theprocessor 1020 to display the image signal processed by the image signalprocessor 1060.

The audio signal processor 1070 may process audio data. The audio signalprocessor 1070 may perform various processes such as one or more ofdecoding, amplifying, noise filtering, etc. on the audio data. Also, theaudio signal processor 1070 may be controlled by the processor 1020 toperform functions corresponding to those of at least one of theelevation perception generator and stage expander 510 and the soundequalizer applier 520 of FIG. 5.

The audio output interface 240 may be controlled by the processor 1020to output the audio signal included in the broadcast signal obtainedthrough the tuner 1010, the audio signal input through the communicator1030 or the inputter/outputter 1050, and the audio signal stored in thememory 1090. The audio output interface 240 may include at least one ofa speaker, a headphone output terminal, or S/PDIF output terminal. Theaudio output interface 240 is controlled by the processor 1020 to outputthe audio signal processed by the audio signal processor 1070.

The user inputter 1080 may be a component (e.g., an interface) throughwhich data for controlling the image and audio processing apparatus 1000may be input by the user. For example, the user inputter 1080 mayinclude at least one of a keypad, a dome switch, a touch pad, a jogwheel, or a jog switch, etc., but is not limited thereto.

Also, the user inputter 1080 may be an element of the control device101, or an element of the image and audio processing apparatus 1000. Theuser inputter 1080 may include a function of obtaining the user inputtransferred from the control device 101.

The block diagrams of the image and audio processing apparatuses 100,700, and 1000 shown in FIGS. 2, 7, and 10 are for an embodiment. Eachcomponent in the block diagram may be combined, added, or omittedaccording to specification of the image and audio processing apparatus100, 700, or 1000 that is actively implemented. For example, ifnecessary, two or more components may be combined as one or onecomponent may be divided as two or more components. Also, functions foreach block are to explain the embodiment and each specific operation ordevice do not limit the scope of the disclosure.

As described above, according to one or more embodiments, when a userreproduces content, at least one of image quality and sound of thecontent may be automatically controlled taking into account at least oneof the image signal and audio signal in units of scenes, and thus,visibility may be improved based on characteristics of the content andrendered sound may be provided to increase user's immersion in thecontent. Also, according to one or more embodiments, based on the resultof analyzing each of the image signal and the audio signal whencontrolling the image signal and the audio signal, it is possible tofaithfully provide the content intended by the manufacturer to the user.Also, according to one or more embodiments, a consistent visibility ofthe content may be provided based on peripheral illuminance. Also,according to one or more embodiments, a consistent visibility and soundof the content may be provided based on the peripheral environment.

The image and audio processing apparatus and the operating methodthereof according to the embodiments may be implemented in the form of acomputer-readable storage medium including computer-executableinstructions, such as computer-executable program modules. Thecomputer-readable storage medium may be any available medium that may beaccessed by a computer, and includes volatile and non-volatile media andremovable and non-removable media. Also, the computer-readable storagemedium may include both a computer storage medium and a communicationmedium. The computer storage medium may include volatile andnon-volatile media and removable and non-removable media that areimplemented using any method or technology for storing information, suchas computer-readable instructions, a data structure, a program module,or other types of data. The communication medium typically includescomputer-readable instructions, a data structure, a program module, orother data of modulated data signal such as carrier waves, or othertransmission mechanisms, and includes an arbitrary information transfermedium.

In addition, the terms such as “ . . . unit”, etc. provided herein maybe realized by a hardware component such as a processor or a circuit,and/or a software component executed by a hardware component such as aprocessor.

Also, the image and audio processing apparatus and the operating methodthereof according to the embodiment may be implemented as a computerprogram product including a recording medium having a program storedtherein, wherein the program for performing an operation of obtainingmultiple language text, and operations of obtaining vector valuesrespectively corresponding to words included in the multiple languagetext, converting the obtained vector values into vector valuescorresponding to target language, and obtaining target language textbased on the converted vector values.

The above description is provided for the purpose of illustration, andit would be understood by those skilled in the art that various changesand modifications may be made without changing technical conception andessential features of the disclosure. Thus, the embodiments set forthherein or shown above are to be interpreted in an illustrative andnon-limiting sense. For example, each component described to be of asingle type can be implanted in a distributed manner. Likewise,components described to be distributed can be implanted in a combinedmanner.

The scope of the disclosure is defined by the following claims ratherthan by the detailed description of the embodiments. It shall beunderstood that all modifications and embodiments of the disclosureconceived from the meaning and scope of the claims and their equivalentsare included in the scope of the disclosure.

What is claimed is:
 1. An image and audio processing apparatuscomprising: a display; an audio output interface; a memory storing oneor more instructions; and a processor configured to execute the one ormore instructions stored in the memory to: detect characteristicinformation of an image signal of content based on analyzing the imagesignal in units of the content, detect characteristic information of anaudio signal of the content based on analyzing the audio signal in theunits of the content, and control the image signal in the units of thecontent based on the characteristic information of the image signal andthe characteristic information of the audio signal to output the imagesignal through the display, and control the audio signal in the units ofthe content based on the characteristic information of the image signaland the characteristic information of the audio signal to output theaudio signal through the audio output interface.
 2. The image and audioprocessing apparatus of claim 1, wherein the units of content are unitsof scenes.
 3. The image and audio processing apparatus of claim 2,wherein the processor is further configured to execute the one or moreinstructions to: control at least one of a black equalizer function or acontrast and color saturation function with respect to the image signalin the units of the scenes, based on the characteristic information ofthe image signal and the characteristic information of the audio signal,and control at least one of an elevation perception generation and stageexpansion function or a sound equalizer application function withrespect to the audio signal in the units of the scenes, based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal.
 4. The image and audio processingapparatus of claim 2, wherein the characteristic information of theimage signal comprises at least one of brightness information, colorinformation, contrast information, or motion information of the imagesignal, and wherein the characteristic information of the audio signalcomprises at least one of location information of an audio object, voiceinformation, music information, or sound effect information.
 5. Theimage and audio processing apparatus of claim 2, further comprising: anilluminance sensor configured to detect illuminance information withrespect to outside of the image and audio processing apparatus, whereinthe processor is further configured to execute the one or moreinstructions to, based on the image signal being controlled based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal, control at least one of brightness,contrast, or color saturation of the image signal based on theilluminance information.
 6. The image and audio processing apparatus ofclaim 2, wherein the processor is further configured to execute the oneor more instructions to, based on a learning model that includes one ormore neural networks, analyze the image signal and the audio signal ofthe content in the units of the scenes, detect the characteristicinformation of the image signal and the characteristic information ofthe audio signal, and control the image signal and the audio signal. 7.An operating method of an image and audio processing apparatus, theoperating method comprising: analyzing, by the image and audioprocessing apparatus, an image signal of content in units of the contentand detecting characteristic information of the image signal; analyzing,by the image and audio processing apparatus, an audio signal of thecontent in the units of the content and detecting characteristicinformation of the audio signal; and controlling, by the image and audioprocessing apparatus, the image signal in the units of the content tooutput the image signal through a display of the image and audioprocessing apparatus, based on the characteristic information of theimage signal and the characteristic information of the audio signal; andcontrolling, by the image and audio processing apparatus, the audiosignal in the units of the content to output the audio signal through anaudio output interface of the image and audio processing apparatus,based on the characteristic information of the image signal and thecharacteristic information of the audio signal.
 8. The operating methodof claim 7, wherein the units of content are units of scenes.
 9. Theoperating method of claim 8, further comprising: controlling at leastone of a black equalizer function, or a contrast and color saturationfunction with respect to the image signal in the units of the scenes,based on the characteristic information of the image signal and thecharacteristic information of the audio signal; and controlling at leastone of an elevation perception generation and stage expansion functionor a sound equalizer application function with respect to the audiosignal in the units of the scenes, based on the characteristicinformation of the image signal and the characteristic information ofthe audio signal.
 10. The operating method of claim 8, wherein thecharacteristic information of the image signal comprises at least one ofbrightness information, color information, contrast information, ormotion information of the image signal, and the characteristicinformation of the audio signal comprises at least one of locationinformation of an audio object, voice information, music information, orsound effect information.
 11. The operating method of claim 8, furthercomprising, based on the image signal being controlled based on thecharacteristic information of the image signal and the characteristicinformation of the audio signal, at least one of brightness, contrast,or color saturation of the image signal is controlled based onilluminance information with respect to outside of the image and audioprocessing apparatus, the illuminance information being detected by theimage and audio processing apparatus.
 12. A computer-readable recordingmedium having recorded thereon program instructions which are executableby a processor for performing an operating method of a image and audioprocessing apparatus, the operating method comprising: analyzing animage signal of content in units of the content and detectingcharacteristic information of the image signal; analyzing an audiosignal of the content in the units of the content and detectingcharacteristic information of the audio signal; and controlling theimage signal in the units of the content to output the image signalthrough a display of the image and audio processing apparatus, based onthe characteristic information of the image signal and thecharacteristic information of the audio signal; and controlling theaudio signal in the units of the content to output the audio signalthrough an audio output interface of the image and audio processingapparatus, based on the characteristic information of the image signaland the characteristic information of the audio signal.